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Method and Apparatus for Automated Software Testing 

The present invention relates to software testing and in particular to unit testing 
software during its operation. The invention can be applied advantageously, but not 
5 exclusively, to software produced using object oriented programming languages such 
as C + + , Corba or Java. 

Automated testing of software during its development is known. The tests are 
designed as part of a software development process and these are then programmed 
1 0 into specialised test tools and executed automatically. Many tools are commercially 
available to support this type of software development technique. 

Software that checks itself during operation is also known and has been developed 
and applied widely. This may involve checking pre and post-conditions or assertions 
1 5 and looking for exceptions at appropriate points in the software during its normal 
execution (See "Self Testing Systems" - M Aylett and P Utton, BT Technology 
Journal 1992). 

Known testing systems enable end-to-end tests to be run on operational software 
20 systems in order to test out the operation of individual facilities. However, there are 
currently no testing systems that easily enable low level tests to be run on a fully 
integrated and operational system. These tests are often termed "unit tests" and are 
applied directly to one or more individual units of code (e.g. a function, method, 
module or agent). This is in contrast to end-to-end tests of a system that run from a 
25 system or user interface. Unit tests are currently run manually or automatically during 
development before integration. 



According to the present invention there is provided a method of testing an 
operational integrated software system, said system comprising a plurality of 
30 software elements, said method comprising the steps of: 
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b) associating a set of test criteria with each registered element of 
software; 

c) selecting an element registered in the registry and testing the element 
in accordance with the associated set of test criteria; and 

5 d) capturing the results of the testing of the element and comparing them 

to the associated test criteria. 

This provides the advantage of enabling unit testing to be carried out on an 
integrated software system during its operation that allows quick identification of 
10 latent or newly introduced faults in the software. 

Figure 1 is a schematic representation of a computer loaded with software 
embodying the present invention; 

Figure 2 is a functional block diagram of the program elements that comprise the 
1 5 software indicated in Figure 1 ; 

Figure 3 is a flow diagram illustrating part of the processing of the software shown in 
Figure 2; 

Figures 4a and 4b are tables illustrating the data structures used and created by the 
program elements shown in Figure 2; and 
20 Figure 5 is a flow diagram showing a further part of the processing of the software 
shown in Figure 2. 

Figure 1 illustrates a conventional computer 101 such as a PC, running a 
conventional operating system 103 such as Windows and having a number of 

25 resident application programs 105 such as a word processing program, a network 
browser and e-mail program or a database management program. The computer 101 
also includes a software development application program 107 that enables the user 
to write and compile new programs and a testing program 109 that enables testing to 
be carried out on programs. The computer 101 is also connected to a conventional 

30 disc storage unit 111 for storing data and programs, a keyboard 1 13 and mousel 15 
for allowing user input and a printer 117 and display unit 119 for providing output 
from the computer 101. The computer 101 also has access to external networks (not 
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In conventional object oriented programming the programs are divided into conceptual 
sub-units called objects. Each object carries out predetermined functions much in the 
same way that a sub-routine might in conventional programming. Objects carry out 
5 processing of data and may co-operate with other objects to carry out some 
functions. Such co-operation is carried out via interfaces between the objects called 
arguments that are provided for passing commands, requests and data between th 
objects. 

10 Each object is categorised into a class of objects. In fact, it is the class of an object 
that determines the functions and performance of an object. An object itself is an 
embodiment (or instance) of the class and can be created to carry out its function 
and then deleted once the function is complete. The creation of an object for a given 
class is carried out under the control of a constructor algorithm. In addition, the 

1 5 corresponding destructor for each class is arranged to remove the entry when th 
corresponding object is deleted. 

Each object comprises one or more methods. Each method is a subroutine that 
together with other methods provides the functions of the object itself. Methods may 
20 co-operate with other objects to carry out functions/processing on behalf of the 
method. The methods are also defined by the class of the object as are the 
arguments of the object. 

In summary, objects are functional units of software code whose functions are 
25 defined by the class of which a given object is an instance. Objects can have a 
number of states that change depending on the object's interaction with other 
objects or data. The combined interaction of the objects that make up a computer 
program provide the functions of the program itself. 

30 With reference to figure 2, the testing program 109 comprises five main components, 
a tester 201, an object registry 203, a report generator 205, a test criteria store 207 
and a parser 209. The tester 201 carries out the testing of each object in the 
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generator 205. The object registry 203 provides the tester 201 with a list of the 
objects that form part of the program at any given time (as noted above, objects may 
be created and destroyed during the operation of a program). The test criteria store 
207 is used to hold the data and/or instructions necessary to test each of the objects 
5 registered in the object registry 203. In the present embodiment the data and/or 
instructions held in the test criteria store 207 are immediately usable by the tester 
201. However, in some cases the data may be coded using a scripting language. In 
this case the parser 209 would be used to convert the data/instruction into a form 
usable by the tester 201 . The functions and interactions of the five main components 
10 will be described in further detail below. 

Figure 2 also shows a program object 21 1 undergoing testing by the tester 201 . The 
object 211 is a standard object but has three additional areas of functionality that 
allow it to interact automatically with the testing program 109. The added 
15 functionality is provided in the present embodiment by two special methods 213, 
215 added to each class definition used in the program under test and by additions to 
the functionality of the constructor and destructor algorithms for the program. 

With reference to figure 3, the constructor is arranged, on the instantiation of an 
20 object for a given class, to create an entry in the object registry 203 for the new 
object (see step 301 of chart C). Then, at step 303, the constructor enters the 
identification for the object in its entry in the registry 203 (each object, when it is 
constructed by the constructor, is assigned a unique identifier). At step 305, the 
class type of the object is entered in the entry for the object and at step 307 the 
25 corresponding class name is entered. After step 307 the registration process is 
completed and the constructor algorithm ends its processing. 

As noted above, when an object is no longer required it is deleted by a destructor 
algorithm. In the present embodiment, the destructor algorithm is also arranged to 
30 carry out the steps shown in chart D of Figure 3. At step 309 the destructor 
algorithm identifies the entry in the registry 203 that corresponds to the object being 
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With reference to figure 4a, each class of object has a test criteria file that is entered 
into the test criteria store 207 when the first object of that class is entered in the 
object registry 203. The criteria are created during the design and implementation of 
the computer program under test and their precise construction is dependent on the 
5 testing methods being used. In the present embodiment, an entry is made in the store 
207 for each class 401. For every class, an entry 403 is made for each method 
within the class. For each method 403, a definition of the input 405 to the method, 
the output 407 from the method, the start state 409 of the object when the method 
is performed and the end state 41 1 of the object on completion of the method is 
10 entered in the store 207. 

The operation of the tester 201 will be described now with reference to Figure 5 in 
which at step 501 the tester 201 awaits a command to commence testing. In the 
present embodiment the command is given by a user. Once the command has been 

15 received then, at step 503, the tester 201 chooses the class of object to be tested 
from the registry 203. In the present embodiment, the system responds to a user 
command to commence testing and the chooses a method at random. However, the 
command or choice of method could be produced randomly, in accordance with a 
predefined testing plan or in response to requests or events from other objects or 

20 programs. 

At step 505 the tester 201 uses the first special method 213 to determine the 
number of methods in the chosen object. The method 213 returns data, as shown in 
Figure 4b, describing the class of the object 413, identifications 415 of each of the 
25 methods in the object and a description 417 of the arguments for each of the 
method. At step 507, using the class identification returned by the method 213, the 
tester 201 identifies the appropriate test criteria from the test criteria store 207 and 
at step 509 runs the chosen method against the identified test criteria. 

30 At step 511, the tester 201 uses the second special method 215 to capture the 
results of the test run on the method. The precise data that is captured is determined 
by the test criteria and may include the output data from the tested method, the 
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methods that the chosen method interacted with as a result of the test. At step 513, 
the test data collected in the previous step is compared to the test criteria and the 
results of the comparison are passed to the report generator 205 for inclusion in a 
test report. After step 513, the tester returns to step 501 to await a further test 
5 instruction. 

The tester program 201 is designed to carry out its testing procedures on a program 
while the program is in operation. In some operating systems the testing program 
201 could be arranged to run as a background process or be arranged to operate 
10 when there is a predetermined amount of spare processor resource available. 

As will be understood by those skilled in the art, in some systems it may be 
necessary to include means for preventing changes to the run-time environment being 
made during the testing of a software element. These may be in the form of run-time 
15 test switches that are similar in function to a debug compiler switch. In some 
systems it may be necessary to include a means to restore the state of any persistent 
variables (variables that retain state after execution) affected by the tests. This can 
be performed by taking a copy of the persistent variables before a test and restoring 
them afterwards. 

20 

It will also be clear to those skilled in the art that the system under test could 
distributed in nature. For example, testing could be carried out over a network and 
units of code distributed across many computers. Also, the testing system can be 
used by developers during the design and build of a software system or be provided 
25 as part of the functionality of programs that are ready for use. 

The tester program is preferably written in the same language as the program that it 
is testing. However, although the embodiment above describes the testing of an 
object oriented programming language, it will be understood by those skilled in the art 
30 that the principles of the invention are also applicable to other programming 
languages. Other such languages may be modular programming languages (such as 
Modula-2) 
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understood that the term "object" used in the this description is to be construed 
broadly so as to cover functions, methods, modules or agents. 

As will be understood by those skilled in the art, the tester program 109 can be 
5 contained on various transmission and/or storage mediums such as a floppy disc, CD- 
ROM, or magnetic tape so that the program can be loaded onto one or more general 
purpose computers or could be downloaded over a computer network using a suitable 
transmission medium. 

10 Unless the context clearly requires otherwise, throughout the description and the 
claims, the words "comprise", "comprising" and the like are to be construed in an 
inclusive as opposed to an exclusive or exhaustive sense; that is to say, in the sense 
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CLAIMS 

1 . A method of testing an operational integrated software system, said system 
comprising a plurality of software elements, said method comprising the steps of: 

5 

a) automatically registering each active element of software in a registry; 

b) associating a set of test criteria with each registered element of 
software; 

c) selecting an element registered in the registry and testing the element 
10 in accordance with the associated set of test criteria; and 

d) capturing the results of the testing of the element and comparing them 
to the associated test criteria. 

2. A method according to Claim 1 in which each element of software is arranged 
1 5 to automatically register an identification of itself in the registry. 

3. A method according to Claims 1 or 2 in which each element of software is 
arranged to capture the results of its testing. 

20 4. A method according to any of Claims 1 to 3 further comprising the step of 
automatically providing a report on the results of the testing. 

5. A method according to any preceding claim in which the test criteria are 
defined using a scripting language and said method further comprises the step of 

25 parsing the test criteria to convert them into a form for testing against. 

6. An apparatus for testing an operational integrated software system, said 
system comprising a plurality of software elements, said apparatus comprising: 

30 a) means for the automatic registration of each active element of 

software; 
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c) means for selecting a registered element of software and testing the 
element in accordance with the associated test criteria; and 

d) means for comparing the results of the testing of the element against 
the associated test criteria. 

5 

7. An apparatus according to Claim 6 in which each element of software is 
provided with means for automatically registering itself. 

8. A method according to Claim 6 or 7 in which each element of software is 
10 provided with means for capturing the results of its testing. 

9. An apparatus according to any of Claims 6 to 8 further comprising means for 
producing a report of the results of testing an element of software. 

15 10. An apparatus according to any of Claims 6 to 9 in which the test criteria are 
defined using a scripting language and the apparatus further comprises means for 
parsing the test criteria to convert them into a form for testing against. 

11. A data carrier loadable into a computer and carrying instructions for causing 
20 the computer to carry out the method according to Claim 1 . 

1 2. A data carrier loadable into a computer and carrying instructions for enabling 
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MACHINERY., vol. 37, no. 9, September 1994 (1994-09), pages 87-101, 
XP000485275 ASSOCIATION FOR COMPUTING MACHINERY. NEW 
YORK., US ISSN: 0001-0782 



ANNEX TO SECTION V 

1 As explained below, independent claim 1 appears to be new and inventive. 

Claim 1 relates to a method of testing an operational integrated software system. 
Each active element of software is automatically registered and associated with a 
set of test criteria. An element of software is selected from the registered elements 
and tested in accordance with the associated set of test criteria. The results of the 
testing are captured and compared to the associated test criteria. 

The ISR cites only the document D1. D1 is, therefore, considered the closest prior 
art document. It describes a method of testing an object-oriented system. In D1 a 
test driver generates a set of test criteria from a test specification belonging to an 
element of software. The test driver tests the element of software in accordance 
with the generated test criteria and captures and evaluates the results. However, 
D1 merely describes how a single element of software is tested. It is neither 
described nor suggested in D1 which or how elements of software are selected 
from the object oriented system for testing. In contrast thereto, claim 1 defines a 
method for testing all active elements of software of the operational integrated 
software system by automatically registering each active element of software and 
selecting from these registered elements an element of software for test. 

Therefore, the subject-matter of claim 1 is new and inventive and meets the 
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2 Claims 2-5 are dependent on claim 1 and add further features to the inventive 
method of claim 1. Therefore, the subject-matters of claims 2-5 are new and 
inventive and meet the requirements of Article 33(2) and 33(3) PCT 

3 Each of the system claims 6-1 0 has a counterpart in the method claims 1-5. 
Therefore, the same observations as for method claims 1-5 apply for system 
claims 6-10. Thus, the subject-matters of claims 6-10 are new and inventive and 
meet the requirements of Article 33(2) and 33(3) PCT. 

4 Claim 1 1 defines a data carrier carrying instructions for causing the computer to 
carry out the method according to claim 1. Claim 12 defines a data carrier 
carrying instructions for enabling the computer to provide the system according to 
claim 6. Therefore, the same observations as for claims 1 and 6 apply for 
claims 11 and 12. Thus, the subject-matters of claims 11 and 12 are new and 
inventive and meet the requirements of Article 33(2) and 33(3) PCT. 

ANNEX to SECTION VII 

1 To meet the requirements of Rule 5. 1(a)(ii) PCT, the closest prior art document 
D1 should have been identified in the description and the relevant background art 
disclosed therein should have been briefly discussed. 

2 The independent claims should have been drafted in the two-part form in 
accordance with Rule 6.3(b) PCT with those features known from prior art 
document D1 being placed in the preamble and the remaining features being 
included in the characterizing part. 

3 The features of the system claims should have been provided with reference signs 



